Continue with last model, we consider the model (plug in all possible ${\displaystyle c_i}$ 's): $$\begin{array}{rl}{y}_t=& {\beta }_0+{\beta }_1(t-1)+{\beta }_2\mathrm{ReLU}(t-2)+\cdots \\ \text{(1)}& & +{\beta }_{n-1}\mathrm{ReLU}(t-(n-1))+{\epsilon }_t.\end{array}$$
Note that it is different from our last model: $$\begin{array}{rl}{y}_t=& {\beta }_0+{\beta }_1(t-1)+{\beta }_2\mathrm{ReLU}(t-c_1)\\ \text{(2)}& & +\cdots +{\beta }_{k+1}\mathrm{ReLU}(t-c_k)+{\epsilon }_t.\end{array}$$
The new model does not have parameters ${\displaystyle c_k}$, so it is linear (discussed here). Also (2) will be used with a small ${\displaystyle k}$, while (1) can have large ${\displaystyle n}$. In short, (1) is a high-dimensional linear regression model, and (2) is a low-dimensional nonlinear regression model.

(1) has ample parameters so it is a flexible model.

1 Parameter Interpretation in (1)

Let ${\displaystyle {\mu }_t}$ denote the deterministic part in (1) (without ${\displaystyle {\epsilon }_t}$): $$\begin{array}{rl}{\mu }_t=& {\beta }_0+{\beta }_1(t-1)+{\beta }_2\mathrm{ReLU}(t-2)+\cdots \\ & +{\beta }_{n-1}\mathrm{ReLU}(t-(n-1)).\end{array}$$ Then (1) can be rewritten as $$y_t={\mu }_t+{\epsilon }_t,{\epsilon }_t\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\sigma }^2).$$

The noise term can have several interpretations.

• When we see ${\displaystyle {\mu }_t}$ as trend, ${\displaystyle {\epsilon }_t}$ can be seen as random fluctuations.
• When we see ${\displaystyle {\mu }_t}$ as actual data, ${\displaystyle {\epsilon }_t}$ can be seen as measurement error.

Plug in ${\displaystyle t=1}$, we have $${\beta }_0={\mu }_1.$$ Plug in ${\displaystyle t=2}$, we have $${\beta }_1={\mu }_2-{\mu }_1.$$ Similarly, $${\beta }_t=({\mu }_{t+1}-{\mu }_t)-({\mu }_t-{\mu }_{t-1}).$$

If we use this model to represent logarithm of population, then

• ${\displaystyle 100{\beta }_t}$ represents difference between percentage change from ${\displaystyle t}$ to ${\displaystyle t+1}$, and percentage change from ${\displaystyle t-1}$ to ${\displaystyle t}$, for ${\displaystyle t\ge 1}$;
• ${\displaystyle {\beta }_0}$ represents the scale of data
• ${\displaystyle 100{\beta }_1}$ represents the percentage change: let ${\displaystyle P_t=\exp ({\mu }_t)}$, then $${\beta }_1=\log P_2-\log P_1=\log \frac{P_2}{P_1}\approx \frac{P_2}{P_1}-1.$$

So they have different scales.

2 Parameter Estimation

2.1 Unregularized MLE

For linear regression model (1), we can estimate as usual by MLE: $$\sum _{t=1}^n(y_t-{\beta }_0-{\beta }_1(t-1)-{\beta }_2\mathrm{ReLU}(t-2)-\cdots -{\beta }_{n-1}\mathrm{ReLU}(t-(n-1)){)}^2.$$
We have $${\hat{\sigma }}_{\mathrm{MLE}}=\sqrt{\frac{\mathrm{RSS}}{n}}.$$
However now the number of data points is equal to coefficients, so ${\displaystyle \mathrm{RSS}=0,\hat{\sigma }=0}$. The unbiased estimate ${\displaystyle \sigma }$ will not exist because ${\displaystyle \sqrt{\frac{\mathrm{RSS}}{n-p}}}$ and ${\displaystyle n=p}$.

So traditional estimates overfit the data.

2.2 Regularization

Now we add regularization. ${\displaystyle {\hat{\beta }}_{\mathrm{ridge}}(\lambda )}$ (ridge estimator) is the minimizer of $$\sum _{t=1}^n{(y_t-{\beta }_0-{\beta }_1(t-1)-\sum _{j=2}^{n-1}\mathrm{ReLU}(t-j))}^2+\lambda \sum _{j=2}^{n-1}{\beta }_j^2.$$
We also have LASSO estimator ${\displaystyle {\hat{\beta }}_{\mathrm{lasso}}(\lambda )}$: $$\sum _{t=1}^n{(y_t-{\beta }_0-{\beta }_1(t-1)-\sum _{j=2}^{n-1}\mathrm{ReLU}(t-j))}^2+\lambda \sum _{j=2}^{n-1}|{\beta }_j|.$$

Correspondingly, plug in ${\displaystyle \mu }$, RIDGE goes to Hodrick-Prescot Filter: $$\sum _{t=1}^n(y_t-{\mu }_t{)}^2+\lambda \sum _{t=2}^{n-1}[({\mu }_{t+1}-{\mu }_t)-({\mu }_t-{\mu }_{t-1}{)}^2].$$
LASSO goes to ${\displaystyle L_1}$ -trend Filter: $$\sum _{t=1}^n(y_t-{\mu }_t{)}^2+\lambda \sum _{t=2}^{n-1}|({\mu }_{t+1}-{\mu }_t)-({\mu }_t-{\mu }_{t-1})|.$$

We can rewrite this into matrix form: $$||y-X\beta |{|}^2+\lambda \sum _{t=2}^{n-1}{\beta }_j^2.$$ Take ${\displaystyle J=\mathrm{diag}(0,0,1,\cdots ,1)}$, then $$\mathrm{\nabla }(||y-X\beta |{|}^2+\lambda \sum _{t=2}^{n-1}{\beta }_j^2)=-2X^{\mathrm{T}}y+2X^{\mathrm{T}}X\beta +2\lambda J\beta .$$ Let it be ${\displaystyle 0}$, we have $$\begin{array}{}\text{(2.1)}& {\hat{\beta }}^{\mathrm{ridge}}(\lambda )=(X^{\mathrm{T}}X+\lambda J{)}^{-1}{X}^{\mathrm{T}}y.\end{array}$$ Compared with regular ${\displaystyle \hat{\beta }=(X^{\mathrm{T}}X{)}^{-1}{X}^{\mathrm{T}}y}$, the only difference is the ${\displaystyle \lambda J}$ term.

For LASSO, ${\displaystyle f(\beta )=(y-\beta {)}^2+\lambda |\beta |}$.

3 Cross Validation for Selecting ${\displaystyle {\displaystyle \lambda }}$

We can use cross validation to select ${\displaystyle \lambda }$. First split the total set ${\displaystyle T=\{1,\cdots ,n\}}$ into ${\displaystyle T_{\mathrm{train}},T_{\mathrm{test}}}$. (say 80%: 20%). For this split, fit the model in ${\displaystyle T_{\mathrm{train}}}$, and obtain ${\displaystyle {\hat{\beta }}_{\mathrm{train}}^{\mathrm{ridge}}(\lambda )}$ as minimizer of $$\begin{array}{rl}& \sum _{t\in T_{\mathrm{train}}}(y_t-{\beta }_0-{\beta }_1(t-1)-{\beta }_2\mathrm{ReLU}(t-2)-\cdots -{\beta }_{n-1}\mathrm{ReLU}(t-(n-1)){)}^2\\ & +\lambda ({\beta }_2^2+\cdots +{\beta }_{n-1}^2),\end{array}$$ and ${\displaystyle {\hat{\beta }}_{\mathrm{train}}^{\mathrm{LASSO}}(\lambda )}$ as $$\begin{array}{rl}& \sum _{t\in T_{\mathrm{train}}}(y_t-{\beta }_0-{\beta }_1(t-1)-{\beta }_2\mathrm{ReLU}(t-2)-\cdots -{\beta }_{n-1}\mathrm{ReLU}(t-(n-1)){)}^2\\ & +\lambda (|{\beta }_2|+\cdots +|{\beta }_{n-1}|).\end{array}$$
Using these estimates, predict ${\displaystyle y_t}$ for ${\displaystyle t\in T_{\mathrm{test}}}$: $$\begin{array}{r}{\hat{y}}_t^{\text{ridge}}(\lambda )={\hat{\beta }}_{\text{train},0}^{\text{ridge}}(\lambda )+{\hat{\beta }}_{\text{train},1}^{\text{ridge}}(\lambda )(t-1)+{\hat{\beta }}_{\text{train},2}^{\text{ridge}}(\lambda )\mathrm{ReLU}(t-2)+\cdots +{\hat{\beta }}_{\text{train},n-1}^{\text{ridge}}(\lambda )\mathrm{ReLU}(t-(n-1)),\\ {\hat{y}}_t^{\text{lasso}}(\lambda )={\hat{\beta }}_{\text{train},0}^{\text{lasso}}(\lambda )+{\hat{\beta }}_{\text{train},1}^{\text{lasso}}(\lambda )(t-1)+{\hat{\beta }}_{\text{train},2}^{\text{lasso}}(\lambda )\mathrm{ReLU}(t-2)+\cdots +{\hat{\beta }}_{\text{train},n-1}^{\text{lasso}}(\lambda )\mathrm{ReLU}(t-(n-1)).\end{array}$$
Denote $$\begin{array}{r}{\text{Test-Error}}^{\mathrm{ridge}}(\lambda )=\sum _{t\in T_{\mathrm{test}}}(y_t-{\hat{y}}_t^{\mathrm{ridge}}(\lambda ){)}^2,\\ {\text{Test-Error}}^{\mathrm{lasso}}(\lambda )=\sum _{t\in T_{\mathrm{test}}}(y_t-{\hat{y}}_t^{\mathrm{lasso}}(\lambda ){)}^2.\end{array}$$
Going over all splits, we can have the total test error: $$\begin{array}{r}{\text{AllSplit-Test-Error}}^{\mathrm{ridge}}(\lambda )=\sum _{\text{all splits}}{\text{Test-Error}}^{\mathrm{ridge}}(\lambda ),\\ {\text{AllSplit-Test-Error}}^{\mathrm{lasso}}(\lambda )=\sum _{\text{all splits}}{\text{Test-Error}}^{\mathrm{lasso}}(\lambda ).\end{array}$$
We apply this to a set of candidate ${\displaystyle \lambda }$ values and choose the optimal ${\displaystyle \lambda }$ that minimizes the all split test error.

A common split for test set may be, ${\displaystyle \{5k+i,k\in \mathbb{N}\}}$ for ${\displaystyle i=1,2,3,4,5}$.